Detection of 3D Pose of a TEE Probe in X-ray Medical Imaging

ABSTRACT

Pose of a probe is detected in x-ray medical imaging. Since the TEE probe is inserted through the esophagus of a patient, the pose is limited to being within the esophagus. The path of the esophagus is determined from medical imaging prior to the intervention. During the intervention, the location in 2D is found from one x-ray image at a given time. The 3D probe location is provided by assigning the depth of the esophagus at that 2D location to be the depth of the probe. A single x-ray image may be used to determine the probe location in 3D, allowing for real-time pose determination without requiring space to rotate a C-arm during the intervention.

BACKGROUND

The present embodiments relate to detection of a probe position fromx-ray imaging. Two-dimensional (2D) x-ray imaging (e.g. fluoroscopy) isroutinely used for interventional cardiac surgery. x-ray imaging andtransesophageal echocardiography (TEE) provide complementary informationduring the cardiac surgery. x-ray imaging is used to monitorinterventional devices (e.g., catheter and TEE probe), andthree-dimensional (3D) TEE is used to visualize soft tissue. To fullyutilize complementary information from both modalities, the coordinatesystems of the x-ray and ultrasound are registered or aligned. Detectingthe 3D pose of a TEE probe from x-ray images enables the fusion of x-rayand ultrasound images. With an accurately estimated 3D pose of the TEEprobe, measurements or annotations in the ultrasound image or the x-rayimage can be transferred to the coordinate system of the other modality.

A challenge in 3D pose detection from single view x-ray image is theambiguity of depth, which causes largely inaccurate estimation of theobject's movement along the viewing axis of the C-arm detector of thex-ray system. Conventional methods to resolve the depth ambiguitytypically involve acquiring at least two x-ray images of the TEE probefrom different angles (e.g., 30 degrees apart) during the procedure orintervention. However, a bi-plane system to acquire two x-ray imagessimultaneously is not commonly available due to increased equipmentcost, and the additional radiation dose associated with bi-planeacquisition is undesired. For a mono-plane system, once theinterventional procedure starts, it is inconvenient and may beimpossible to rotate the C-arm to acquire two images at different anglesduring the procedure because of the density of equipment in theoperating room. In addition, real-time TEE probe pose estimation fordynamic fusion of ultrasound and x-ray images becomes impossible when aC-arm needs to be rotated between different angles in order to acquirethe x-ray images used to estimate the pose at each given time.

BRIEF SUMMARY

By way of introduction, the preferred embodiments described belowinclude methods, systems, instructions, and computer readable media fordetection of a probe pose in x-ray medical imaging. Since the TEE probeis inserted through the esophagus of a patient, the pose is limited tobeing within the esophagus. The path of the esophagus is determined frommedical imaging prior to the intervention. During the intervention, thelocation in 2D is found from one x-ray image at a given time. The 3Dprobe location is provided by assigning the depth of the esophagus atthat 2D location to be the depth of the probe. A single x-ray image maybe used to determine the probe location in 3D, allowing for real-timepose determination without requiring space to rotate a C-arm during theintervention.

In a first aspect, a method is provided for detection of a probe pose inx-ray medical imaging. A three-dimensional path of an esophagus of apatient is determined from medical imaging prior to an intervention onthe patient. An orientation and a first location of a trans-esophagealechocardiographic (TEE) probe is detected during the intervention on thepatient. The detection is from a single x-ray image. The x-ray image isacquired during the intervention, and the first location is in twodimensions with ambiguity along a third dimension corresponding to aview direction from an x-ray source used for the single x-ray image. Theambiguity along the third dimension is resolved with thethree-dimensional path, providing a second location in the threedimensions. Coordinates of the x-ray source are aligned with the TEEprobe based on the second location.

In a second aspect, a method is provided for detection of a probe posein x-ray medical imaging. A trajectory along which a probe is to beinserted into a patient is determined. The trajectory is determined inthree-dimensions relative a patient. A location in two dimensions isdetected from an individual x-ray image of the patient with the probewithin the patient. The location in three dimensions is detected fromthe detection of the location in the two dimensions from the individualx-ray image and the trajectory. Fused medical imaging is performed withthe probe and x-ray imaging based on the location in the threedimensions.

In a third aspect, a system is provided for detection in medicalimaging. A transesophageal echocardiography imaging system includes atransducer for imaging with ultrasound from within a patient. An x-raysystem is configured to acquire a sequence of x-ray images with an x-raysource in one position relative to the patient during an intervention.The x-ray images represent the patient and the transducer over time. Animage processor is configured to detect, separately for each of thex-ray images, a three-dimensional location of the transducer relative tothe x-ray source. The three-dimensional location is detected fromproximity of (a) a line from the transducer as detected in the x-rayimage to the position of the x-ray source to (b) a trajectory of anesophagus of the patient.

The present invention is defined by the following claims, and nothing inthis section should be taken as a limitation on those claims. Furtheraspects and advantages of the invention are discussed below inconjunction with the preferred embodiments and may be later claimedindependently or in combination.

BRIEF DESCRIPTION OF THE DRAWINGS

The components and the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of the invention.Moreover, in the figures, like reference numerals designatecorresponding parts throughout the different views.

FIG. 1 is a block diagram of one embodiment of a system for posedetection in medical imaging;

FIG. 2 is a flow chart diagram of one embodiment of a method fordetection of a probe pose in x-ray medical imaging;

FIG. 3 illustrates one embodiment for determining a trajectory of anesophagus;

FIG. 4 illustrates probe pose parameters according to one embodiment;

FIG. 5 illustrates an example of depth ambiguity based on detection ofprobe location in one x-ray image; and

FIG. 6 illustrates resolution of the depth ambiguity in the example ofFIG. 5 using the trajectory of the esophagus.

DETAILED DESCRIPTION OF THE DRAWINGS AND PRESENTLY PREFERRED EMBODIMENTS

Depth ambiguity in 3D pose detection of a TEE probe at a given time isresolved using a single-view x-ray image. The single-view x-ray imageand a 3D trajectory of the esophagus are used to resolve the depthambiguity so that the 3D pose of the TEE probe may be accuratelyestimated. The human esophagus is a narrow tubular structure. Since aTEE probe is put into the esophagus for imaging, the anatomy of theesophagus (e.g., the centerline) provide cross-sectional locationconstraints for the TEE probe.

Being able to accurately register the TEE probe using a single-viewx-ray image may dramatically improve user experience and may enabledynamic fusion between the echo model and x-ray in real-time. Sincex-ray imaging from different directions is not needed during theintervention or treatment procedure, a more seamless workflow for TEEprobe tracking and fusion of information between x-ray imaging andultrasound imaging is provided.

The fusion of x-ray and ultrasound imaging assists many interventionalapplications, especially for complicated catheter-based structural heartdisease (SHD) therapy (e.g., trans-catheter aortic valve repair (TAVR),mitral valve repair (MVR), or left atrial appendage closure (LAAC)). Thealignment of the coordinate systems may be used in other cardiac ornon-cardiac situations or procedures.

FIG. 1 shows a system 10 for pose detection in medical imaging. Thesystem 10 estimates the pose, such as at least 3D location, of the probe18 from an individual or single x-ray image acquired during interventionand a path of the esophagus. The 3D location of the probe 18 in thex-ray system 16 coordinate space is used for fusion imaging and/oralignment of coordinates.

The system 10 includes a memory 12, a transesophageal echocardiography(TEE) imaging system 14 with an image probe 18, an x-ray system 16, animage processor 26, and a display 28. Additional, different, or fewercomponents may be provided. For example, a network or network connectionis provided, such as for networking with a medical imaging network ordata archival system. As another example, a preoperative imaging system,such as a computed tomography or magnetic resonance imaging system, isprovided. In another example, a user interface is provided.

The memory 12, image processor 26 and/or display 28 are part of amedical imaging system, such as the x-ray system 16, TEE imaging system14, or other system. Alternatively, the memory 12, image processor 26,and/or display 28 are part of an archival and/or image processingsystem, such as associated with a medical records database workstationor server. In other embodiments, the memory 12, image processor 26,and/or display 28 are a personal computer, such as desktop or laptop, aworkstation, a server, a network, or combinations thereof. The memory12, image processor 26, display 28, and/or memory 12 may be providedwithout other components for implementing the method.

The memory 12 is a graphics processing memory, a video random accessmemory, a random access memory, system memory, random access memory,cache memory, hard drive, optical media, magnetic media, flash drive,buffer, database, combinations thereof, or other now known or laterdeveloped memory device for storing data or video information. Thememory 12 is part of an imaging system, a computer associated with theimage processor 26, a database, another system, a picture archivalmemory, or a standalone device.

The memory 12 stores data representing a scan region, at differenttimes, of a patient. The region is a two or three-dimensional region.The region is of any part of the patient, such as a chest, thorax,abdomen, leg, head, arm, or region that is a combination thereof. Thedata is ultrasound, x-ray, and/or other image data. The data includesx-ray information representing the probe 18 while the probe 18 is withinthe patient. The data may represent the patient without the probe 18(i.e., while the probe is not within the patient).

The data is from scanning the region by any medical imaging modality.Any type of data may be used, such as medical image data (e.g.,ultrasound, x-ray, computed tomography (CT), magnetic resonance imaging(MRI), or positron emission tomography). In one embodiment, the datarepresenting the patient volume is ultrasound data and x-ray data. Thedata represents the patient prior to, during, and/or after treatment.For example, the x-ray data is acquired during treatment to guidepositioning of the probe and a treatment device (e.g., ablationcatheter), and the ultrasound data is also provided during treatment toguide the treatment relative to the soft tissue of the patient. Othermedical imaging data (e.g., x-ray, MRI or CT) is acquired prior totreatment or intervention in order to determine the 3D location of theesophagus.

Image data is data that can be used to generate an image, pixels valuesto be displayed, pixel values that were displayed, or other frames ofdata representing the region of the patient at a given time. The imagedata may be frames of DICOM data or frames of data generated along anyportion of a data processing path of an imaging system. A sequence offrames of data is acquired, such as acquiring fluoroscopy images overtwo or more heart cycles at any frame rate (e.g., 10-20 frames persecond). Alternatively, one or any number of frames of data are acquiredbased on trigger events rather than an on-going sequence.

The memory 12 alternatively or additionally stores pose information,such as position, orientation, and/or scale of the probe 18. 2D positionand/or 3D position may be stored. Any parameters, thresholds,machine-learnt classifiers, templates, selections, tracking information,transforms, and/or other calculated information for aligning thecoordinate systems may be stored.

The memory 12 or other memory is alternatively or additionally acomputer readable storage medium storing data representing instructionsexecutable by the programmed processor 26 for detecting probe pose froman x-ray image. The instructions for implementing the processes, methodsand/or techniques discussed herein are provided on non-transitorycomputer-readable storage media or memories, such as a cache, buffer,RAM, removable media, hard drive or other computer readable storagemedia. Non-transitory computer readable storage media include varioustypes of volatile and nonvolatile storage media. The functions, acts ortasks illustrated in the figures or described herein are executed inresponse to one or more sets of instructions stored in or on computerreadable storage media. The functions, acts or tasks are independent ofthe particular type of instructions set, storage media, processor orprocessing strategy and may be performed by software, hardware,integrated circuits, firmware, micro code and the like, operating alone,or in combination. Likewise, processing strategies may includemultiprocessing, multitasking, parallel processing, and the like.

In one embodiment, the instructions are stored on a removable mediadevice for reading by local or remote systems. In other embodiments, theinstructions are stored in a remote location for transfer through acomputer network or over telephone lines. In yet other embodiments, theinstructions are stored within a given computer, CPU, GPU, or system.

The TEE imaging system 14 is a medical diagnostic ultrasound imagingsystem built for TEE or used for TEE. In alternative embodiments, theultrasound system is for imaging with any of various probes, such as acatheter, handheld, or other probe. Using ultrasound, the TEE imagingsystem 14 scans a plane, planes, and/or volume of the patient from theprobe 18. The probe 18 includes a transducer, such as an array ofpiezoelectric elements, for converting between acoustic and electricalenergies for scanning the patient.

The probe 18 is an endocavity probe for insertion into the throat of thepatient. By positioning the probe 18 in the esophagus of the patient,the probe 18 may be used to scan the heart or other cardiac region ofthe patient. In alternative embodiments, the probe 18 is a catheter forinsertion into the circulatory system of the patient. The probe 18 ispositioned within the patient for ultrasound imaging. The position isspatially limited, such as being in the esophagus or a vessel. Thescanning is performed using the probe 18 while the probe 18 is in thepatient.

The x-ray system 16 is any now known or later developed x-ray system,such as a C-arm x-ray, fluoroscopy, angiography, or other x-ray system.The x-ray system 16 includes an x-ray source and detector separated by aregion into which the patient is positioned. Using a C-arm or othersupporting structure, the source and detector are positioned relative tothe patient. x-rays generated by the source pass through the patient andthe probe 18 if within the patient, and the detector detects theattenuated x-rays. The C-arm may position the source and detector to beat different angles relative to the patient. During an intervention andsince single x-ray images may be used for estimating pose over time, theC-arm may keep the source and detector at a given angle relative to thepatient. The source and detector are kept at one position during theintervention, but may be moved to other positions before theintervention. Alternatively, the source and detector are moved duringthe intervention.

By transmitting x-rays through the patient to the detector, a projectionimage is provided. Any tissue, bone, the probe 18, and/or other medicaldevice along the path of travel of the x-ray beam interacts with thex-rays, causing a detectable difference in intensity at the detector.Since each pixel or location of the detector represents an accumulationof responses along the path of travel, the x-ray image is a projectionimage of the region.

The x-ray image may be responsive to the probe 18. For example, duringsurgery or other intervention, the probe 18 may be positioned within thepatient for imaging the cardiac system. One or more fluoroscopy imagesare generated in real-time during the procedure. A sequence offluoroscopy images over multiple (e.g., three or more) heart cycles isacquired. The fluoroscopy images represent the patient and/or probe 18in the patient for one or different phases in any one or multiple heartcycles. The frames of the fluoroscopy sequence are acquired andprocessed for detection within seconds of acquisition (e.g., real-timeor on-line detection) or are stored and later retrieved for detection.

The image processor 26 is a general processor, central processing unit,control processor, graphics processor, digital signal processor,three-dimensional rendering processor, image processor, applicationspecific integrated circuit, field programmable gate array, digitalcircuit, analog circuit, combinations thereof, or other now known orlater developed device for detecting pose of the probe 18 and/oraligning coordinate systems of the TEE imaging system 14 and the x-raysystem 16. The image processor 26 is a single device or multiple devicesoperating in serial, parallel, or separately. The image processor 26 maybe a main processor of a computer, such as a laptop or desktop computer,or may be a processor for handling some tasks in a larger system, suchas in an imaging system. The image processor 26 is configured byinstructions, design, firmware, hardware, and/or software to be able toperform the acts discussed herein.

The acts 30-38 of FIG. 2 or sub-sets thereof are implemented by theimage processor 26. Interaction with the TEE system 14 and x-ray system16 are used to acquire the data representing the patient and/ortransducer for determining the pose of the transducer. As compared tothe specifics discussed below for FIG. 2, the operation of the imageprocessor 26 is now described in general.

The image processor 26 is configured to detect, separately for each ofthe x-ray images of a sequence, a 3D location of the transducer relativeto the x-ray source. The 3D location is detected from proximity of (a) aline from the transducer as detected in the x-ray image to the positionof the x-ray source to (b) a trajectory of an esophagus of the patient.The trajectory is provided to or determined by the image processor 26.The image processor 26 finds the closest points of line (a) with thetrajectory (b) as the 3D location. By repeating for each individualx-ray image, dynamic or real-time 3D location detection is provided(i.e., closest points over time are used).

The image processor 26 may determine further pose parameters. Inaddition to the 3D location, the orientation of the transducer at the 3Dlocation is determined. Based on image processing, the direction atwhich the transducer faces for scanning (i.e., normal to a center of anemitting face of the transducer) and/or the direction a tip or end ofthe probe 18 is facing are determined. The x-ray image is processed todetermine the orientation. Like the 3D location, this orientation poseinformation may be determined for each time represented by each x-rayimage. The image processor 26 may estimate scale as another poseparameter.

The display 28 is a monitor, LCD, projector, plasma display, CRT,printer, or other now known or later developed devise for outputtingvisual information. The display 28 receives images, graphics, or otherinformation from the image processor 26, memory 12, TEE system 14,and/or x-ray system 16.

One or more x-ray images representing a probe position relative to apatient region are displayed. The location of the medical device (e.g.,probe 18) is or is not highlighted, marked by a graphic, or otherwiseindicated on the x-ray image. For example, an image includesfluoroscopic information showing the location of a probe 18. Where asequence of images is displayed, the location of the probe 18 is shownin each of the images through the sequence. One or more ultrasoundimages may also be shown. The alignment or position of the ultrasoundimages relative to the x-ray images is output, represented on thedisplay, or incorporated into the displayed images as fusion imaging. Alocation of an annotation or selection in one image may be shown inanother image based on the aligned coordinate systems (i.e., based onthe known position of the probe 18 in the x-ray system 16 coordinatespace). Alternatively, the ultrasound and fluoroscopy images are alignedand fused into one image (e.g., ultrasound image overlaid on part or allof the fluoroscopy images) using the alignment based on the identified3D location.

FIG. 2 shows one embodiment of a method for detection of a probe pose inx-ray medical imaging. Pre-intervention medical imaging is used todetermine a path of a patients esophagus. During intervention, a 3Dlocation of the probe at a given time is estimated from a 2D locationdetected in one x-ray image and the path of the esophagus. For othertimes during the intervention, other individual x-ray images and thesame path of the esophagus are used. In other embodiments, other pathrestrictive (e.g., tubular or spatially constraining) anatomy may beused instead of the esophagus.

The method is implemented by the system 10 of FIG. 1 or a differentsystem. For example, the x-ray system 16 or other imaging systemperforms a scan prior to intervention. The image processor 26 or othersystem, with or without user input, performs act 30 using the results ofthe scan. The x-ray system 16 performs a scan after intervention begins,and the image processor 26 performs act 32 using the results of theintervention scan. The image processor performs acts 34 and 36 based onthe output of acts 30 and 32. The TEE system 14 performs a scan duringthe intervention, and the image processor 26 performs act 38 based onthe TEE scan, the x-ray scan, and the results of act 36. The imageprocessor 26 interacts with the display 28 for performing act 38.

The acts are performed in the order shown (e.g., top to bottom ornumerical order) or a different order. Additional, different, or feweracts may be provided. For example, acts 36 and/or act 38 are notperformed. As another example, acts 32-38 are performed repetitively,such as repeating 10-20 times a second during an intervention.

For performing acts 30, 32, and 38, medical imaging data is acquired.The medical imaging data is one or more frames of data from scanning apatient, such as x-ray, CT, MRI, C-arm CT, and/or ultrasound data. Eachframe represents a 2D or 3D region of the patient at a given time orover a period. The data is processed or used as the data is acquired.Alternatively, the data is stored and later loaded and processed.

In act 30, x-ray, C-arm CT, CT, MRI, or other medical imaging is used.For act 32, x-ray imaging is used. The x-ray system scans a patientwhile the probe is within the patient. The probe may also be used toscan the patient, such as with ultrasound for use in act 38. For posedetection, the frames of data from the x-ray system are used with thetrajectory determined in act 30 from other or the same modality ofmedical imaging. The x-ray scanning is repeated any number of times atany rate. For example, a sequence of frames representing the patient andprobe at different times is acquired. The sequence is acquired at 10-20frames per second, but may have a different rate. The sequence isacquired over seconds or minutes.

In act 30, the image processor determines a 3D path of an esophagus of apatient. The determination of the path from imaging data is automatic,such as not using any user input other than to activate and/or selectdata to be used. The image processor applies segmentation, filtering(e.g., machine-learnt classification), or other image process toidentify locations of the esophagus represented in the medical imagingdata. Alternatively, a manual or semi-automatic process is used. Forexample, the user traces the esophagus in one or more images. As anotherexample, the user selects an esophagus location as a seed point, and theimage processor determines the location of the esophagus based on theimage data and the seed point.

The 3D path is determined from medical imaging prior to an interventionon the patient. During the intervention, the TEE probe is used forimaging soft tissue to guide the intervention or treatment. Theintervention begins by inserting a treatment device into the patient,such as a catheter for ablation. The patient is sedated and placed on anexamination table or gurney. Once the patient is ready, the TEE probeand treatment device are inserted into the patient in any order. Thisinsertion is the beginning of the intervention. Alternatively, theintervention begins once the treatment device is positioned in the organwith the location to be treated (e.g., in the heart). In yet anotheralternative, the intervention begins when the TEE probe is positioned sothat the location to be treated may be imaged. The 3D path is determinedprior to beginning the intervention. In an alternative embodiment, the3D path is determined prior to use to determine the 3D location of theprobe regardless of whether the intervention has begun.

The 3D path is determined as a trajectory along which a probe is to beinserted into a patient. Any range of the esophagus may be included inthe trajectory, such as from a top of the torso (e.g., shoulder) to thestomach or a point below the heart if the patient were standing. Thetrajectory is determined in three-dimensions relative the patient. In aworld coordinate system or a coordinate system of the x-ray system, thepath is found. The path is defined by sampling locations in 3D, such asx, y, z coordinates. Alternatively, the path is found by a line fittingwhere the line may be represented by discrete coordinate values.

The path is determined as a tube. For example, an inner, outer, or bothinner and outer tissue of the esophagus are detected. In anotherexample, a centerline of the esophagus is determined as the path. Basedon detection of the esophagus tissue, a shrinking or other centerlineoperation is used to determine the centerline as the path.

In one embodiment represented in FIG. 3, the 3D trajectory of theesophagus is extracted based on imaging after the patient is on thetable under general anesthesia. Before the intervention starts, the TEEprobe is inserted into the esophagus and placed deeper than the heart,such as near the entrance of the stomach. With the probe in thisposition, the patient is scanned from different directions with thex-ray source. The C-arm moves the x-ray source and detector to differentangles relative to the patient. FIG. 3 shows projection images generatedfrom scanning form two different angles. Any angles may be used, such as45 or 90 degrees apart. The x-ray images are of the thorax or chestregion of the patient.

The esophagus is located in each of x-ray scan images resulting from thescanning from different directions. For example, thresholding, templatematching, or segmentation is applied to find the TEE probe (e.g.,elongated body of the TEE probe) in the esophagus. The centerline of theesophagus is then determined from the detected TEE probe. FIG. 3 showsdetection of the centerline in each of the x-ray images. Thesecenterlines provide 2D distribution of the esophagus as viewed from twodifferent directions. In another approach, the esophagus is determinedwithout using the TEE probe to increase contrast.

The 3D path of the esophagus is reconstructed from the locations of theesophagus in each of the scan images. The 2D centerlines viewed fromdifferent directions geometrically relate to each other. Byreconstructing this relationship, the 3D path is determined as shown inFIG. 3. The 2D centerlines from two different angles are combined toreconstruct the 3D centerline as the 3D trajectory of the esophagus.

Because determining the path of the esophagus is performed only once andbefore the interventional procedure starts, rotating the C-arm toacquire two or more x-ray images from different angles may be moreeasily managed and acceptable to clinicians. In alternative embodiments,the path is determined multiple times. The paths are averaged or usedfor different phases of the cardiac cycle.

In another embodiment, pre-operative 3D imaging is used to determine the3D path of the esophagus. Using x-ray (e.g., C-arm CT), MRI, or CT, datarepresenting a volume of the patient including the esophagus isacquired. This pre-operative volume may be acquired the same day as or adifferent day as the intervention. The esophagus is located in thevolume using image processing and/or manual input (e.g., tracing).

To relate the pre-operative volume to the coordinate system of the x-raysystem used in the intervention, the patient is scanned from differentdirections with the x-ray source without the TEE probe being in thepatient. After the patient is on the table under general anesthesia forthe intervention but before inserting the TEE probe into the patientsesophagus, two or more x-ray images are acquired from different angles,such as 45 or 90 degrees apart. The chest or thorax region is scannedfor each of the x-ray images.

The x-ray images are spatially registered with the pre-operative volumerepresenting the patient. The volume is projected to model the x-rayprojection images. The projection is performed from different anglesand/or scales. Each projection is then fit with the x-ray images. Anymeasure of similarity may be used, such as a cross-correlation orminimum sum of absolute differences. The angles and scales with theprojection most similar to the x-ray images from the different anglesare determined.

These angles and scales for the different directions relate thecoordinate system of the pre-operative volume to the x-ray system. The3D path of the esophagus from the pre-operative volume is thentransformed to the coordinate system of the x-ray system. The 3D path isdetermined from the esophagus as labeled in the pre-operative volume andthe spatially registering. The registration brings the extracted 3Dtrajectory of the esophagus into the patient coordinate system orcoordinate system of the x-ray system.

In yet another embodiment, the x-ray system of the intervention is usedto acquire the volume, so registration may be avoided. A 3D volumerepresenting the patient is acquired using the x-ray source. Anintra-operative C-arm CT volume is acquired prior to the intervention.The patient is sedated and positioned on the table in preparation forthe intervention. Prior to the intervention, a C-arm CT scan isperformed, resulting in a volume representing the patient.

The 3D path of the esophagus as represented in the 3D volume isdetermined. The volume is used for the extraction of the 3D trajectoryof the patients esophagus. The esophagus is found by tracing, seededimaging processing, thresholding, segmentation, filtering,classification, or other approach. In this case, the C-Arm CT volume isby default registered properly with the patient so that an additionalregistration does not need to be performed. The 3D path is in thecoordinate system of the x-ray system.

In act 32, the image processor detects a pose of the TEE probe from anx-ray image acquired during the intervention on the patient. The pose isa position with or without orientation and/or scale. FIG. 4 shows anexample probe and two vectors for orientation. The center represents alocation in three dimensions, such as x, y, z location. The “front”vector represents an orientation of the emitting face of the transducerin the probe head. The “tip” vector represents an orientation of the tipof the TEE probe or the direction of travel of the TEE probe through theesophagus. The front and tip vectors are restricted to be orthogonal,but other relationships may be used. Other location, orientation, and/orscale parameterizations may be used. For example, orientation isparameterized as three Euler angles (e.g., yaw, roll, and/or pitch).

The pose of the TEE probe is detected using any of various imageprocesses. For example, a machine-learnt classifier or detector isapplied to the x-ray frame of data to detect the pose. Using inputfeatures, the machine-learnt detector finds the location, orientation,and/or scale based on a learned relationship between the input featurevectors and pose. A hierarchy of machine-learnt detectors may be used,such as finding location, then orientation, and then scale with thesequence of detectors boot-strapped together. As another example,template matching is used to detect the pose. Representative templatesof the head of the TEE probe at different orientations are correlatedwith the x-ray image at different positions. The location with thegreatest correlation indicates location. The orientation of the templatewith the greatest correlation indications the orientation. The scale ofthe located and oriented template with the greatest correlationindicates the scale. Combinations of approaches may be used. Any nowknown or later developed approach to detect pose from the x-ray imagemay be used.

A single x-ray image is used to determine the pose at a given time. Thex-ray image is acquired during the intervention. After the TEE probe isinserted for use to scan the tissue and guide the treatment or afterbeginning the intervention, an x-ray image is acquired to determine thelocation of the TEE probe in the patient. The detection of the pose isperformed for the given x-ray image independently of or without alsousing another x-ray image. X-ray images from different times during theintervention and/or from different angles are not used in the detectionof the pose. In alternative embodiments, a pose from a previousdetection is used to guide or provide a starting point for finding thepose in the current detection, but the x-ray images are from the sameviewing direction and the detection is performed in just the currentx-ray image.

While a single image and/or view direction is used, the detection isrepeated using other x-ray images. As each x-ray image in a sequence isacquired, a pose is determined for each of the individual one of thex-ray images. This provides a sequence of poses over time.

In a given or individual x-ray image, the location is provided in twodimensions. The location of the TEE probe is detected as a point in theprojection plane. The TEE probe has a point parameterized as thelocation of the transducer or head of the TEE probe. In the example, ofFIG. 4, the center of the head is used. The tip, center of thetransducer array, or other location on the TEE probe may be used as thelocation.

The orientation, scale, and 2D location are detectable from the singlex-ray image. This 2D location is ambiguous in the third dimension. FIG.5 shows detection of the location in two dimensions with ambiguity alonga third dimension. The projection of the x-ray image collapses the thirddimension corresponding to a view direction from an x-ray source. Whilethe location in the 2D projection plane of the x-ray image is detected,the location of the TEE probe along the depth relative to the x-raysource is not known, is uncertain, or may not be accurately detected byimage processing the x-ray image. The location of the TEE probe along aline 62 extending from the x-ray source to the 2D location in the x-rayimage represents possible depths for the TEE probe. The depth ambiguitymeans that the position of the TEE probe is uncertain along the viewingdirection.

In act 34, the image processor resolves the ambiguity along the thirddimension with the 3D path of the esophagus. The z or depth location inthe Cartesian system of the x-ray system is determined so that a depth(z) of the 3D location (x, y being in the projection plane) isestimated. Where the Cartesian or other coordinate system being useddoes not align with the x-ray system, the projection plane provideslocation with ambiguity along one direction and the ambiguity along thisone direction is resolved with the 3D path. The resolution of thelocation provides the location in three dimensions. The 2D location fromthe projection x-ray image is resolved into a 3D location, includingdepth along the line from the x-ray source to the 2D location. Forexample, the center of the TEE probe detected in the 2D x-ray image orprojection plane is resolved to a location in three dimensions (locationin two dimensions and a location in the third dimension).

The location in three dimensions is detected from the detected locationin two dimensions in the individual x-ray image and from the trajectory(e.g., path of the esophagus). The depth ambiguity is resolved bycombining the detected 3D pose (e.g., orientation+2D location with depthambiguity) with the 3D trajectory of the esophagus.

FIG. 6 shows one embodiment of resolving the depth. The point 60 isdetermined as the 3D location, including the depth. This pointrepresents the 3D location of the TEE probe, such as being a 2D locationof the TEE probe in the projection plane or x-ray image 64 plus a depthrelative to the x-ray source. To find the point 60, a line 62 passingthrough the x-ray source and the 2D location in the projection plane orx-ray image 64 is calculated. The point 60 is calculated as the locationalong the line 62 having a shortest distance to the 3D path 66 of theesophagus. For example, the line 62 connecting the center of the TEEprobe detected from the x-ray image 64 and the center of the x-raysource provides possible positions of the center of the TEE probe. Thecenter of the TEE probe should also be within the esophagus. Combiningthis information, the point along the line 62 with the shortest distanceto the 3D trajectory (i.e., path 66) of the esophagus is selected as thecenter of the TEE probe. Since the 3D trajectory is provided in thex-ray or patient coordinate system, the location on the line 62 closestto the path 66 indicates the depth of the TEE probe. The shortestdistance is found as a Euclidian distance, but other measures may beused. This closest approach between the line 62 and the 3D path 66resolves the depth ambiguity.

The location in three dimensions is determined based on detection in asingle or individual x-ray image. By detecting the probe location in the2D x-ray image, the 3D location may be resolved using the trajectory.The detection for 2D location is performed only in one x-ray image for agiven time during the intervention. The 3D path 66 of the esophagus isused to resolve depth ambiguity without needing an image from adifferent direction, allowing real-time 3D location detection. The same3D path 66 may be used with other detections in other x-ray images 64for other times during the intervention. In other embodiments, more thanone x-ray image 64 may be used for detection of location at a giventime.

Referring again to FIG. 2, the image processor aligns coordinatesbetween the x-ray source and the TEE probe based on the 3D location ofthe TEE probe in the x-ray coordinate system. A spatial transformbetween the x-ray source and the TEE probe provides the alignment. The3D location is a pose or part of a pose of the probe relative to thex-ray system. The 3D location provides a transform relating the twosystems. The position of the ultrasound image in the coordinate systemof the probe may be related to or aligned with the position of the x-rayimage in the coordinate system of the x-ray system.

The alignment may be used to inform the physician of probe positionrelative to the patient and/or ultrasound image relative to the x-rayimage. The alignment allows for fusion of different types of images,fusion by indication of the spatial relationship (e.g., position,orientation, and/or scale) between the types of images, and/or fusion bycross-reference between modalities of annotation, marking, or detection.

The pose of the probe in each frame of x-ray data provides the spatialrelationship between the x-ray system and the ultrasound system of theprobe. In one embodiment, a 3D point Q_(TEE) in the ultrasoundcoordinate system is projected in the x-ray image at 2D image pointQ_(Fluoro). The relationship is represented as:

Q _(fluoro) =P _(int) P _(ext)(R _(TEE) ^(W) Q _(TEE) +T _(TEE) ^(W))

where P_(int) is x-ray or camera internal projection matrix, P_(ext) isx-ray camera external matrix that transforms a point from a worldcoordinate to camera coordinate system. The internal and external relatethe movable C-arm mounted x-ray and detector space to the world space ofthe x-ray system. The internal and external matrices are known fromcalibration and C-Arm rotation angles. The ultrasound system, includingthe probe, are also defined in the world space. R_(TEE) ^(W) and T_(EE)^(W) are the rotation and position of the TEE probe in the worldcoordinate system. R_(TEE) ^(W) and T_(TEE) ^(W) are computed from:

R _(TEE) ^(W) =P _(ext) ⁻¹ R _(TEE) ^(C)

T _(TEE) ^(W) =P _(ext) ⁻¹ T _(TEE) ^(C)

where R_(TEE) ^(C) and T_(TEE) ^(C) are the rotation and position of theTEE probe in the x-ray coordinate system. R_(TEE) ^(C)=(θ_(z), θ_(x),θ_(y)), and T_(TEE) ^(C)=(x; y; z). By detection of the pose, therelationship may be determined. Other transforms, such as between theinternal x-ray space and the ultrasound space without reference to theworld space, may be used.

The poses over time are used to align the coordinates of the x-raysystem with the probe over time. The alignment occurs for each time.Alternatively, an average transform is calculated and used forsubsequent registration, transformation, or conversion.

In act 38, the image processor generates a fused image from the x-rayimage and an image generated using the TEE probe. The 3D location isused to create the transform, which may then be used to association alocation in one image to a corresponding location in another image(e.g., selection of a tissue location in an ultrasound image results inindication of the tissue location in the x-ray image). Other fusedmedical imaging, such as displaying adjacent x-ray and ultrasound, maybe used.

In one embodiment, the image processor generates a fused image from oneof the frames of data from the x-ray system and an image generated usingthe TEE probe. Using the alignment, the spatial locations represented byone type of imaging are transformed to coordinates of another type ofimaging. This alignment may be used to create a display with two typesof images shown side-by-side. The perceived relative positioning isbased on the alignment, helping the viewer understand the relationship.Alternatively or additionally, one type of imaging is transformed intothe coordinates of another type and overlaid. For example, a fluoroscopyimage is displayed as a gray-scale image. For a region in the gray-scaleimage, ultrasound data replaces the fluoroscopic data or thefluoroscopic data is overlaid by an ultrasound image. The ultrasoundimage may be in color. Any now known or later developed display ofimages from two types of imaging using relative alignment may be used.In other embodiments, the detected probe may be highlighted in an image,such as by modulating the brightness, overlaying a graphic of the probe,or coloring the pixels of the probe.

As an alternative to fusion of images or as a use for fused imaging, thecoordinate systems of the two systems (e.g., x-ray and ultrasound) arealigned. Landmarks selected by a physician in the ultrasound (3D) aretransferred or relatable to an x-ray image (2D). Those overlaidlandmarks are bio-markers guiding the physician during intervention.While fusion of images may not occur, the images from both types ofimaging and the coordinate alignment assist in guiding the physician asa form of fusion imaging. The fusion imaging uses the transform torelate images, either by overlay, adjacent display with alignment,and/or related marking.

As represented by the arrow from act 38 to act 32, the detection of thepose with 2D location, resolving depth ambiguity to provide 3D location,alignment of the coordinates, and generation of the fused image arerepeated. The detection and resolving of acts 32 and 34 are performedwith a different individual x-ray image. As each x-ray image isacquired, the 3D position of the probe is determined. The sequence ofx-ray images acquired during the intervention are used to determinepose, including 3D position, of the TEE probe over time. For each time,an individual x-ray image is used to find the pose. The trajectory maybe based on other x-ray images or other medical imaging from before theintervention, but each pose determined during the intervention usesdetection of the probe in a single one of the x-ray images of thesequence. Using previous poses to initialize the search for detectionstill provides for detection in that single x-ray image for a givenpose. This may allow for real-time (e.g., 10 frames per second or more)tracking of the probe pose and alignment of the coordinate systems.Different individual frames of x-ray data are used to determine pose atdifferent times.

While the invention has been described above by reference to variousembodiments, it should be understood that many changes and modificationscan be made without departing from the scope of the invention. It istherefore intended that the foregoing detailed description be regardedas illustrative rather than limiting, and that it be understood that itis the following claims, including all equivalents, that are intended todefine the spirit and scope of this invention.

I (WE) CLAIM:
 1. A method for detection of a probe pose in x-ray medicalimaging, the method comprising: determining, from medical imaging priorto an intervention on the patient, a three-dimensional path of anesophagus of a patient; detecting an orientation and a first location ofa trans-esophageal echocardiographic (TEE) probe during the interventionon the patient, the detecting being from a single x-ray image, the x-rayimage acquired during the intervention, and the first location being intwo dimensions with ambiguity along a third dimension corresponding to aview direction from an x-ray source used for the single x-ray image;resolving the ambiguity along the third dimension with thethree-dimensional path, the resolving providing a second location in thethree dimensions; and aligning coordinates of the x-ray source with theTEE probe based on the second location.
 2. The method of claim 1 whereindetermining comprises determining the three-dimensional path of theesophagus as a centerline of the esophagus.
 3. The method of claim 1wherein determining comprises: scanning the patient from differentdirections with the x-ray source while the TEE probe extends in theesophagus past a heart of the patient; locating the esophagus in each ofscan images resulting from the scanning from different directions; andreconstructing the three-dimensional path of the esophagus from thelocations of the esophagus in each of the scan images.
 4. The method ofclaim 1 wherein determining comprises: scanning the patient fromdifferent directions with the x-ray source without the TEE probe beingin the patient; spatially registering scan images resulting from thescanning from different directions with a pre-operativethree-dimensional volume representing the patient; and determining thethree-dimensional path from an esophagus labeled in the pre-operativethree-dimensional volume and the spatially registering.
 5. The method ofclaim 1 wherein determining comprises: acquiring a three-dimensionalvolume representing the patient using the x-ray source; determining thethree-dimensional path as represented in the three-dimensional volume.6. The method of claim 1 wherein detecting comprises detecting the firstlocation as a point in a projection plane of the single x-ray image ofthe TEE probe within the patient and the orientation as a vector of afront and tip of the TEE probe.
 7. The method of claim 1 whereindetecting comprises detecting with a machine learnt detector.
 8. Themethod of claim 1 wherein detecting comprises detecting the firstlocation and orientation in a projection plane of the single x-rayimage.
 9. The method of claim 1 wherein resolving comprises finding apoint along a line that passes through the x-ray source and through thefirst location, the second location being at the point.
 10. The methodof claim 9 wherein resolving comprises selecting the point as being onthe line and having a shortest distance to the three-dimensional path.11. The method of claim 1 wherein detecting provides a center of the TEEprobe as the first location in the two dimensions and wherein theresolving provides the center of the TEE probe as the second location inthe three dimensions, the second location being the first location inthe two dimensions and a position in the third dimension.
 12. The methodof claim 1 wherein aligning comprises calculating a transform betweenthe x-ray source and the TEE probe.
 13. The method of claim 1 furthercomprising repeating detecting and resolving during the interventionwithout repeating the determining.
 14. The method of claim 1 furthercomprising generating a fused image from the x-ray image and an imagegenerated using the TEE probe.
 15. A method for detection of a probepose in x-ray medical imaging, the method comprising: determining atrajectory along which a probe is to be inserted into a patient, thetrajectory being in three-dimensions relative the patient; detecting alocation in two dimensions from an individual x-ray image of the patientwith the probe within the patient; detecting the location in threedimensions from the detection of the location in the two dimensions fromthe individual x-ray image and the trajectory; and performing fusedmedical imaging with the probe and x-ray imaging based on the locationin the three dimensions.
 16. The method of claim 15 wherein determiningthe trajectory comprises determining a path of an esophagus of thepatient, and wherein detecting the location in three dimensionscomprises calculating a point along a line from an x-ray source for theindividual x-ray image to the location in the two-dimensions with aclosest approach to the path.
 17. The method of claim 15 whereindetermining the trajectory comprises determining prior to beginning anintervention guided by imaging with the probe, and further comprisingrepeating the detecting the location in the two dimensions and detectingthe location in the three dimensions with a different individual x-rayimage acquired during the intervention.
 18. A system for detection inmedical imaging, the system comprising: a transesophagealechocardiography imaging system comprising a transducer for imaging withultrasound from within a patient; an x-ray system configured to acquirea sequence of x-ray images with an x-ray source in one position relativeto the patient during an intervention, the x-ray images representing thepatient and the transducer over time; and an image processor configuredto detect, separately for each of the x-ray images, a three-dimensionallocation of the transducer relative to the x-ray source, thethree-dimensional location detected from proximity of (a) a line fromthe transducer as detected in the x-ray image to the position of thex-ray source to (b) a trajectory of an esophagus of the patient.
 19. Thesystem of claim 18 wherein the image processor is further configured todetermine, from each of the x-ray images, an orientation of thetransducer at the respective three-dimensional location.
 20. The systemof claim 18 wherein the image processor is configured to detect thethree-dimensional locations for the x-ray images as a closest point ofthe line with the trajectory.